Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information.

However, [A][B] use similar space-time factorization, in which separate spatial and temporal graph convolutions are performed. Considering this, the novelty of this work is weakened. More experiments on complex human-object interaction datasets, e.g., Charades, would be helpful in showing the scalability of the adopted rigid region-split scheme. It would also be helpful to compare with the existing space-time graphical modeling approaches, e.g., [B][C][33], on such datasets. There are no ablation studies concerning these two modules to shed a light on which part actually brings the performance boost. Studies for analyzing the correlation between performance and number of scales are also missing.

All reviewers agree that the submission should be accepted and appreciate the novelty, clarity and potential for impact.

Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information.

Learning in the space-time domain remains a very challenging problem in machine learning and computer vision. Current computational models for understanding spatio-temporal visual data are heavily rooted in the classical single-image based paradigm. It is not yet well understood how to integrate information in space and time into a single, general model. We propose a neural graph model, recurrent in space and time, suitable for capturing both the local appearance and the complex higher-level interactions of different entities and objects within the changing world scene. Nodes and edges in our graph have dedicated neural networks for processing information.